User Guide-Supported ISP algorithms

Table of Contents [Sticky]

Introduction

PVA ISP provides independent ISP stages and combined stages for NVIDIA Jetson. The C++ examples load examples/configs/params.yaml; applications using the library may set stage properties directly. The YAML keys below describe the example loader and are not a separate stage API. PVA ISP also provides joined kernels and an optional DLUT tone-mapping path. Pipeline selection is made by the application, not by the YAML file.

The images on this page illustrate individual algorithm outputs. The joined and DLUT stages have no separate output images here.

Supported ISP Algorithms

Input Bayer image
Decompanding
Black level correction
Auto white balance
Demosaicing
Global tone mapping
Gamma tone mapping
NV12 output

RGB images below are planar, with full-size red, green, and blue planes. Bayer and RGB16 samples use 16-bit storage. The scene-key side frame is 1×2 float32: [mean_log2_luminance, max_luminance]. Auto white-balance gains use a 1×3 float32 side frame in RGB order. NV12 has a full-size Y plane followed by interleaved UV samples at half vertical resolution; image width and height must be even.

Stage (factory name)	Main input	Main output or side output	Description
`decompand`	Companded Bayer, 10–16 meaningful bits in a 16-bit container	Linear Bayer16	Piecewise-linear expansion.
`black_level`	Bayer16	Corrected Bayer16; optional RGB gains	Optical-black subtraction, range rescaling, per-site digital gains, and optional gain estimation.
`wb_gains`	Bayer16	Unchanged Bayer16; RGB gains	Gain estimation without black-level correction.
`histogram`	Bayer16	256-bin uint32 histogram	Bayer-domain luminance statistics with optional mask.
`white_balance`	Bayer16 and generated or manual RGB gains	Balanced Bayer16	Applies per-color gains.
`demosaic`	Bayer16	Planar RGB16; optional scene key	5×5 Malvar–He–Cutler interpolation.
`convolution`	Planar RGB16	Planar RGB16 and scene key	Per-channel 3×3 spatial filter.
`color_correction`	Planar RGB16	Planar RGB16	3×3 color correction matrix.
`convolution_color_correction`	Planar RGB16	Planar RGB16; optional scene key	Spatial filtering followed by color correction in one PVA stage.
`global_tone_mapping`	Planar RGB16 and scene key	Planar RGB8	Global Reinhard-style tone mapping.
`global_tm_dlut`	Planar RGB16 and host-accessible scene key	Planar RGB8	Global tone mapping with a per-frame 4096-entry display-luminance LUT.
`gamma_tone_mapping`	Planar RGB8	Planar RGB8	Generated or supplied 256-entry gamma LUT.
`nv12_conversion`	Planar RGB8	NV12 (8-bit Y and interleaved UV)	RGB to full-range BT.601-style NV12.
`auto_exposure`	uint32 histogram	`1×2 uint32` exposure and camera gain controls	Host-side PID controller; optional V4L2 writes.
`input_stage`	Companded Bayer	Corrected Bayer16; optional RGB gains	Joined decompand, black level, per-site digital gains, and gain estimation.
`pre_rgb`	Bayer16 and generated or manual RGB gains	Planar RGB16; optional scene key	Joined white balance and demosaic.
`post_rgb`	Planar RGB16 and scene key	NV12	Joined standard tone mapping, gamma, and NV12 conversion.
`post_rgb_dlut`	Planar RGB16 and host-accessible scene key	NV12	Joined DLUT tone mapping, gamma, and NV12 conversion.

Pipeline variants

The isp_minimal example uses separate stages by default:

Decompand → Black Level / WB Gains → White Balance → Demosaic
→ [Convolution] → Color Correction → Global Tone Mapping
→ Gamma Tone Mapping → NV12 Conversion

--join-kernels uses Input Stage → Pre RGB → [Convolution + Color Correction / Color Correction] → Post RGB. The optional --no-convolution removes spatial filtering; --disable-blc omits optical-black correction and per-site digital gains. --gtm-dlut selects global_tm_dlut in the separate graph or post_rgb_dlut in the joined graph. The scene key is supplied by Demosaic or Convolution in the separate graph, and by Pre RGB or Convolution + Color Correction in the joined graph. Histogram and Auto Exposure are available as separate stages; they are not in this isp_minimal image-to-NV12 graph.

Configuration Model

The example YAML has sections for decompand, convolution, color_correction, black_level, digital_gain, white_balance, demosaic, histogram, auto_exposure, gamma_tone_mapping, global_tone_mapping, and debug. The joined stages reuse those values; they have no separate YAML sections. The loader also checks equal, strictly increasing decompand point lists, ordered GTM scale limits, and ordered Auto Exposure limits. The schema requires a 256-entry gamma_tone_mapping.lut when build_lut: false.

Example: examples/configs/params.yaml

# yaml-language-server: $schema=./params.schema.yaml
# Copyright (C) 2026 RidgeRun, LLC (http://www.ridgerun.com)
# All Rights Reserved.
#
# The contents of this software are proprietary and confidential to RidgeRun,
# LLC.  No part of this program may be photocopied, reproduced or translated
# into another programming language without prior written consent of
# RidgeRun, LLC.  The user is free to modify the source code after obtaining
# a software license from RidgeRun.  All source code changes must be provided
# back to RidgeRun without any encumbrance.

# Reference parameter file for the ISP examples.
# This sample matches `params.schema.yaml` and the runtime loader used by:
# - isp_minimal
# - isp_minimal_cuda
# - isp_multicamera
# - isp_app

decompand:
  input_shift: 4
  x_pts: [0, 1098, 1885, 2501, 3062, 3602, 3914, 3975, 4095]
  y_pts: [0, 1107, 3325, 7766, 16652, 34430, 52207, 56648, 65535]

convolution:
  kernel: [1, 2, 1, 2, 4, 2, 1, 2, 1]
  shift: 4

color_correction:
  matrix: [1, 0, 0, 0, 1, 0, 0, 0, 1]
  shift: 0

black_level:
  channel00: 16
  channel01: 16
  channel10: 16
  channel11: 16

digital_gain:
  channel_gains: [1.5, 1.5, 1.5, 1.5]

white_balance:
  u_ref: [0.982, 0.789, 0.668, 0.558, 0.426]
  v_ref: [0.308, 0.382, 0.445, 0.428, 0.678]
  use_calibration: false
  manual_wb: false
  manual_wb_gains: [1.0, 1.0, 1.0]

demosaic:
  bayer_pattern: 0

histogram:
  bit_depth: 16
  use_mask: false

auto_exposure:
  kp_exposure: 40.0
  ki_exposure: 1.0
  kd_exposure: 8.0
  kp_gain: 5000000.0
  ki_gain: 10.0
  kd_gain: 500.0
  reference_mean: 120.0
  dynamic_range: 255.0
  min_exposure: 5000
  max_exposure: 20000
  min_gain: 2000000
  max_gain: 100000000
  camera_index: 0
  enable_v4l2: false

gamma_tone_mapping:
  build_lut: true
  gamma: 2.2
  lut: [
    0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
    16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
    32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
    48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
    64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79,
    80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95,
    96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111,
    112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127,
    128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143,
    144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159,
    160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175,
    176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191,
    192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207,
    208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223,
    224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239,
    240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255
  ]

global_tone_mapping:
  brightness: 0.18
  gtm_scale_min: 0.0001
  gtm_scale_max: 0.01
  gtm_temporal_alpha: 1.0

debug:
  export_scene_key: false
  export_debug_csv: false
  debug_export_frame_index: 0

Decompanding

The stage maps compressed sensor codes to linear 16-bit Bayer values using piecewise-linear control points. x_pts are compressed input codes; y_pts are their corresponding linear output values. The example starts at 0 in both lists and uses nine strictly increasing pairs.

YAML parameter	Sample value	Purpose
`decompand.input_shift`	`4`	Passed to the stage's `shift_bits` property. It identifies known-zero low alignment bits in left-aligned 16-bit input storage; those bits must actually be zero. It is not a request to discard arbitrary meaningful low bits.
`decompand.x_pts`	`[0, 1098, 1885, 2501, 3062, 3602, 3914, 3975, 4095]`	Compressed input-domain control points.
`decompand.y_pts`	`[0, 1107, 3325, 7766, 16652, 34430, 52207, 56648, 65535]`	Linear 16-bit output-domain control points.

The stage uses a 4096-entry sampled LUT plus an endpoint. It interpolates samples between entries for input depths above 12 bits. The input FrameSpec supplies the meaningful bit depth (10–16); bit depth plus shift_bits must not exceed 16, and the control points must fit that logical input range. Out-of-domain codes clamp to the curve endpoints.

Black Level Correction

This stage subtracts configured optical-black offsets at the four positions of the repeating Bayer 2×2 cell, rescales the remaining range, applies the matching per-site digital gain, and clamps to [0, 65535]. It does not measure optical-black pixels itself. The color represented by each position depends on the Bayer pattern.

M = 65535
site = 2 × (row mod 2) + (column mod 2)
output = clamp((input - B) × (M / (M - B)) × digital_gain.channel_gains[site], 0, M)

YAML parameter	Sample value	Purpose
`black_level.channel00`, `channel01`, `channel10`, `channel11`	`16` each	Offsets at even/even, even/odd, odd/even, and odd/odd row/column positions.
`digital_gain.channel_gains`	`[1.5, 1.5, 1.5, 1.5]`	Per-site gains ordered `[channel00, channel01, channel10, channel11]`, combined with the matching site's range rescaling.

The host prepares Q7 correction factors. The black level must stay below 65535, and the Q7 factor must fit the kernel coefficient width. If automatic white balance is selected, this stage also emits [R, G, B] gains based on corrected Bayer data. With manual white balance, the gain side output is skipped.

Sites:

White-Balance Gain Estimation

The wb_gains stage is the alternative gain source when --disable-blc is selected. It passes the Bayer16 image through unchanged and emits a 1×3 float32 RGB gain frame. It requires both five-element calibration vectors even when use_calibration is false. Its scratch arrays support images up to 3840×2160.

Without calibration, the stage computes channel averages and raises weaker channels toward the largest mean. With white_balance.use_calibration: true, it weights tiles using five u_ref = ln(R/G) and v_ref = ln(B/G) reference pairs and blends the resulting gains. Calibration points should represent the expected image population; unsupported or empty channel statistics can produce invalid divisions.

Histogram Generation

The histogram stage converts Bayer16 data into 256 uint32 bins, using one luminance observation per Bayer 2×2 cell. It is a statistics side path rather than an image transformation. Auto Exposure can consume its output.

YAML parameter	Sample value	Purpose
`histogram.bit_depth`	`16`	Logical depth used to shift luminance into the 8-bit bin index; accepted range is 8–16.
`histogram.use_mask`	`false`	Use supplied Q7 weights per Bayer 2×2 cell when true; otherwise weight cells uniformly.

The stage property hist_bins must remain 256 with the PVA kernel. A mask is runtime frame data, not a YAML file path. Each Q7 mask value applies to one Bayer 2×2 cell: 0 excludes, 128 has weight 1, and 256 has weight 2. With no mask, the host creates a uniform mask.

White Balance

White balance applies one gain per Bayer color before demosaicing so neutral scene colors stay neutral under different lighting. The white_balance stage accepts a generated 1×3 float32 gains frame or manual gains. Both green positions use the green gain.

YAML parameter	Sample value	Purpose
`white_balance.u_ref`	`[0.982, 0.789, 0.668, 0.558, 0.426]`	Five log-chromaticity R/G calibration references for automatic estimation.
`white_balance.v_ref`	`[0.308, 0.382, 0.445, 0.428, 0.678]`	Five log-chromaticity B/G calibration references.
`white_balance.use_calibration`	`false`	Enables calibration-weighted gain estimation.
`white_balance.manual_wb`	`false`	Selects manual gains instead of a generated gains frame.
`white_balance.manual_wb_gains`	`[1.0, 1.0, 1.0]`	Manual RGB gains when `manual_wb` is true.

The gain estimator is part of black_level or input_stage when those stages are used, and is a separate wb_gains stage when black-level correction is disabled. The multiplier itself is white_balance in the separate graph or pre_rgb in the joined graph.

Demosaicing

A Bayer pixel contains only one measured color. Demosaicing reconstructs the missing two colors at each pixel. PVA ISP uses fixed 5×5 Malvar–He–Cutler filters: bilinear interpolation is corrected using nearby cross-channel gradients. It is a linear, non-adaptive method; it does not detect edges and switch interpolation directions per pixel. Output is planar RGB16.

YAML parameter	Sample value	Purpose
`demosaic.bayer_pattern`	`0`	Bayer pattern in the input frame specification: 0 BGGR, 1 RGGB, 2 GBRG, 3 GRBG.

With the stage property compute_scene_key enabled before configuration, Demosaic also emits the 1×2 float32 scene key for tone mapping. The side frame must then be bound. This property cannot be changed after configuration because it selects a different PVA executable.

Convolution

The 3×3 convolution filters each RGB16 plane independently and clamps the result to 16 bits. It is used for smoothing or sharpening according to the supplied coefficients. The example uses a weighted smoothing kernel. This stage also emits a scene key measured from the filtered RGB output.

output(x,y,c) = clamp(sum(kernel(i,j) × input(x+i,y+j,c)) >> shift, 0, 65535)

YAML parameter	Sample value	Purpose
`convolution.kernel`	`[1, 2, 1, 2, 4, 2, 1, 2, 1]`	Nine signed coefficients in row-major order.
`convolution.shift`	`4`	Normalizes the accumulated result.

Color Correction

Color correction mixes the channels at each RGB16 pixel using a 3×3 matrix. The coefficients are signed integers; use shift to set their fixed-point scale. The output is clamped to 16 bits. Unlike white balance, this operation can mix different input colors into each output color.

[R', G', B'] = clamp((matrix × [R, G, B]) >> shift, 0, 65535)

YAML parameter	Sample value	Purpose
`color_correction.matrix`	`[1, 0, 0, 0, 1, 0, 0, 0, 1]`	Nine row-major signed coefficients; sample is an identity matrix.
`color_correction.shift`	`0`	Right shift after the color-channel sums.

Global Tone Mapping

Global tone mapping compresses linear RGB16 to display-oriented RGB8. A scene-key frame supplies mean log2 luminance and maximum luminance. The standard kernel derives one exposure scale from the mean and applies a white-point Reinhard-style multiplier equally to R, G, and B, preserving channel ratios. The white point can be smoothed across frames.

YAML parameter	Sample value	Purpose
`global_tone_mapping.brightness`	`0.18`	Target geometric-mean scene brightness.
`global_tone_mapping.gtm_scale_min`	`0.0001`	Minimum exposure scale.
`global_tone_mapping.gtm_scale_max`	`0.01`	Maximum exposure scale.
`global_tone_mapping.gtm_temporal_alpha`	`1.0`	White-point smoothing factor; 1 follows the current frame immediately.

The scale limits must be ordered. The intended temporal-alpha range is 0–1. The standalone standard and DLUT algorithms use different arithmetic paths and are not assumed to produce bit-identical images.

DLUT Global Tone Mapping

global_tm_dlut is the standalone alternative chosen by isp_minimal --gtm-dlut. It has the same primary RGB16 and RGB8 formats and uses the same global_tone_mapping YAML parameters. Its scene-key frame must be host-accessible because the host rebuilds a 4096-entry display-luminance LUT before each frame. The PVA looks up a luminance-dependent multiplier and applies it to all three channels.

The LUT represents 12-bit luminance indices 0–4095. Larger indices use the final entry, so meaningful scene luminance above that range loses distinct mappings. Unlike the standard kernel, DLUT adaptation measures the white point after applying exposure scale (scale × max_luminance). The DLUT and standard outputs need not be bit-exact.

Gamma Tone Mapping

Gamma tone mapping converts planar RGB8 to planar RGB8 through a 256-entry LUT applied to each channel. A generated table uses round(255 × (input/255)^(1/gamma)). Under this convention, gamma greater than 1 brightens midtones; gamma equal to 1 is an identity map.

YAML parameter	Sample value	Purpose
`gamma_tone_mapping.build_lut`	`true`	Generate the table from `gamma`. Set false to use `lut`.
`gamma_tone_mapping.gamma`	`2.2`	Finite positive exponent for generated LUT.
`gamma_tone_mapping.lut`	Identity values 0–255 in the sample	Exactly 256 output bytes, one for each 8-bit input code; required when `build_lut` is false. Ignored while the generated LUT is active.

NV12 Conversion

The standalone stage converts planar RGB8 to NV12 using full-range BT.601-style fixed-point coefficients. Y is generated per pixel; U and V are averaged over 2×2 blocks and interleaved. The stage has no YAML parameters. Even image dimensions are required to preserve the chroma grid.

Joined Stages

The following stages combine operations in fewer PVA passes. Their YAML inputs are the existing sections documented above; applications can also configure their stage properties directly. The existing images illustrate the constituent algorithms and are not claimed as output examples for these joined stages.

Input Stage

input_stage joins decompanding, black-level correction, per-site digital gains, and optional automatic white-balance gain estimation. Its main input is companded Bayer in a 16-bit container; its main output is corrected Bayer16. It uses decompand, black_level, digital_gain, and white_balance calibration values. When manual_wb is false, bind its gains output to pre_rgb; otherwise configure manual gains downstream. The --disable-blc example sets black offsets to zero and all four digital gains to one in this joined stage.

Pre RGB

pre_rgb applies generated or manual RGB white-balance gains to Bayer16 and demosaics the result with the same 5×5 Malvar–He–Cutler method. It produces planar RGB16. Its compute_scene_key property can select a variant that also emits the 1×2 float32 scene key. The property and matching side-frame binding must be set before configuration. The joined example uses Pre RGB's scene key when convolution is disabled.

Convolution + Color Correction

convolution_color_correction joins the 3×3 per-plane convolution and the 3×3 color matrix. It processes planar RGB16 and outputs planar RGB16, taking values from the convolution and color_correction YAML sections. It can optionally compute a scene key from the final color-corrected image; compute_scene_key and its output binding must be set before configuration. This is the scene-key source in the joined example when convolution is enabled. Its color-matrix sum uses signed 32-bit accumulation; sufficiently large coefficients can overflow before the final shift and saturation.

Post RGB

post_rgb joins standard global tone mapping, gamma LUT mapping, and NV12 conversion in one PVA pass. It accepts planar RGB16 plus a scene key and outputs NV12. It uses the global_tone_mapping and gamma_tone_mapping YAML values. The standard white-point adaptation and generated/custom gamma-table rules above also apply. No intermediate RGB8 frames are written between these joined operations.

Post RGB DLUT

post_rgb_dlut is selected by isp_minimal --join-kernels --gtm-dlut. It joins DLUT-based global tone mapping, gamma LUT mapping, and NV12 conversion. It accepts planar RGB16 and a host-accessible scene key, then produces NV12. As in the standalone DLUT path, the host rebuilds the 4096-entry luminance table each frame, and luminance indices above 4095 use its final entry. Its configurable values come from the existing global_tone_mapping and gamma_tone_mapping sections.

Auto Exposure Algorithm

Auto Exposure is a host-side control stage. It consumes a host-accessible uint32 histogram and produces a host-accessible 1×2 uint32 control frame containing [exposure, camera_gain]. It does not process image pixels or run a PVA executable. The histogram must be nonempty; the Histogram stage produces 256 bins.

The histogram weighted mean is compared with reference_mean, using dynamic_range to normalize the error. Separate PID loops update sensor gain and exposure. Gain is adjusted first; exposure is updated when gain reaches a configured limit. The results are clamped to their configured ranges. With enable_v4l2: false, the stage publishes computed controls only. With it true, it also writes the controls to /dev/videoN, using the V4L2 controls named Exposure and Gain where available.

YAML parameter	Sample value	Purpose
`kp_exposure`, `ki_exposure`, `kd_exposure`	`40.0`, `1.0`, `8.0`	Exposure PID coefficients.
`kp_gain`, `ki_gain`, `kd_gain`	`5000000.0`, `10.0`, `500.0`	Camera-gain PID coefficients.
`reference_mean`, `dynamic_range`	`120.0`, `255.0`	Target mean and histogram-index range/error normalizer.
`min_exposure`, `max_exposure`	`5000`, `20000`	Exposure clamps; minimum must not exceed maximum.
`min_gain`, `max_gain`	`2000000`, `100000000`	Camera-gain clamps; minimum must not exceed maximum.
`camera_index`, `enable_v4l2`	`0`, `false`	Select `/dev/videoN` and optionally apply camera controls.

❯

Share This Page

Send Email Using